1. Field of the Invention.
The present invention relates to computation, and more particularly, to encoding inputs to neural network and related distributed memory and computation devices.
2. Description of the Related Art.
Attempts to understand the functioning of the human brain have led to various "neural network" models in which large numbers of neurons are interconnected. These models roughly presume each neuron exists in one of two states (quiescent and firing) with the state determined by the states of connected nuerons (if enough connected neurons are firing, then the original neuron should be in the firing state); and the thrust of the models is to perform computations such as pattern recognition with the neural networks.
J. Hopfield, Neural Networks and Physical Systems with Emergent Collective Computational Abilities, 79 Proc.Natl.Acad.Sci. USA 2554 (1982) describes a neural network model with N neurons each of which has the value 0 or 1 (corresponding to quiescent and to firing), so the state of the network is then a N-component vector V=[V.sub.1,V.sub.2, . . . ,V.sub.N ] of 0's and 1's which depends upon time. The neuron interconnections are described by a matrix T.sub.i,j defining the influence of the j.sup.th neuron on the i.sup.th neuron. The state of the network evolves in time as follows: each neuron i has a fixed threshold .theta..sub.i and readjusts its state V.sub.i randomly in time by setting V.sub.i equal to 0 or 1 depending on whether ##EQU1## is negative or positive. All neurons have the same average rate of readjustment, and the readjustments define a dynamical flow in state space.
With the assumption that T.sub.i,j is symmetric, the potential function ##EQU2## can be used to show that the flow of the network is to local minima of the potential function. Further, with a given set of uncorrelated N-component vectors U.sup.1, U.sup.2, . . . , U.sup.s, a T.sub.i,j can be defined by ##EQU3## and with the thresholds equal to 0, these U.sup.k are the fixed points of the flow and thus stable states of the network. Such a network can act as a content-addressable memory as follows: the memories to be stored in the network are used to construct the U.sup.k and hence T.sub.i,j, so the stored memories are fixed points of the flow. Then a given partial memory is input by using it to define the initial state of the network, and the state will flow usually to the closest fixed point/stable state U.sup.k which is then the memory recalled upon input of the partial memory.
Further analysis and modified network models appear in, for example, J. Hopfield et al, Computing with Neural Circuits: A Model, 233 Science 625 (1986) and J. Hopfield, Neurons with Graded Response Have Collective Computational Properties like Those of Two-State Neurons, 81 Proc.Natl.Acad.Sci. USA 3088 (1984).
D. Willshaw et al, Non-Holographic Associateive Memory, 222 Nature 960 (1969) analyzed the storage capacity of a network of N neurons and concluded that states represented by sparse vectors (the N-component vectors have almost all components equal to 0 which corresponds to almost all neurons quiescent) can be used to store nearly the theoretical maximum amount of information and still have reliable recall if the number of non-zero components is log.sub.2 N.
D. Ackley et al, A Learning Algorithm for Boltzmann Machines, 9 Cognitive Science 147 (1985) describe neural networks with additional adjustment mechanisms for the neurons which analogize thermal fluctuations; this permits escape from local minima of the potential function. However, this disrupts the flow to fixed points for memory recall of the Hopfield type neural networks.
L. Cooper, A Possible Organization of Animal Memory and Learning, Proc.Nobel Symp.Coll.Prop.Phys.Sys. 252 (Academic, New York 1973) observes that the modelling of neural network for animal memory and learning has the problem of mapping events in the animal's environment (i.e., sensory output) to signal distributions in the animal's neurons with the fundamental property of preserving closeness or separateness (in some sense not yet completely defined) of the events. That is, with a vector representation of the neural network states, two events as similar as a white cat and a gray cat should map into vectors which are close to parallel while two events as different as the sound of a bell and the sight of food should map into vectors that are close to orthogonal. Note that standard analysis. such as described in Gonzalez and Wintz, Digital Image Processing (Addison-Wesley 1977), does not use neural network computation and does not have this problem; rather, the standard analysis attempts to extract features and categorize by serial number crunching.
However, the problem of encoding sensor output for neural network input to preserve some sense of closeness is not solved in the known neural networks.